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ABSTRACT 

Just-In-Time Learning (JIT Learning) is a semester-long 
graduate course that teaches corporate trainers and instructional designers 
how to design performance improvement interventions. This course is part of a 
Master’s program in Instructional Technology at San Francisco State 
University. The course has been offered three times and has been completed by 
a total of 38 people. Learners produce two major deliverables for this 
course: a paper describing a ■ performance problem and the design of a system 
to address the problem, and a shorter paper explaining the design strategy. 
Based on final grades for the course, participants in this class can be 
grouped into three broad categories. The A-level performers are 38% of the 
total students, B-level are 17%, and C-level are 45%. An Electronic 
Performance * Support System (EPSS) was designed and implemented to help the B- 
level and C-level people improve their performance. The JIT Learning EPSS was 
designed and developed by Infomark Software Corporation with CGM 
Communications and assistance from several graduate students in the 
Department in the first half of 2001. The JIT Learning EPSS was implemented 
with 15 learners in Fall 2001. For this class the grade distribution was 53% 
in the A-level, 20% in the B-level, and 27% in the C-level. When compared 
with prior classes, all of the B-level performers improved to A-level and 
half of the C-level improved to B-level. A formative evaluation of the system 
was conducted in the winter of 2002. The results suggest that the remaining 
C-level performers do not have a sufficient mental model of the design task. 
The next version of the software will focus on helping users develop and 
assess their mental models before they begin the design task. Includes three 
color figures . (Author/AEF) 



Reproductions supplied by EDRS are the best that can be made 
jfrom the original document. 



cR021782 






PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



G.i._Marks 



Authentic Performance of Complex Problem-Solving Tasks with an EPSS 



. - ■ llMf-ORMATlOh 

^ CENTER (ERIC) 

^ This document has been reproduced as 



□ Minor changes have been made to 

improve reproduction quality. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



By Chet Leighton and Cynthia McCabe 



utMi position or policy. 



Teaching People to Design 

Just-In-Time Learning (ITEC 805) is a semester-long graduate course that teaches corporate trainers and 
instructional designers how to design performance improvement interventions. This course is part of a Master's 
program in Instructional Technology at San Francisco State University. The course has been offered three times and 
in ' has been completed by a total of 38 people. Learners produce two major deliverables for this course-- a paper of 

^ about 20 pages describing a performance problem and the design of a system to address the problem, and a paper of 

about 5 pages explaining the design strategy. 

Q Based on final grades for the course, participants in this class can be grouped into three broad categories. The A- 

^ level performers are 38% of the total students, B-level are 17%, and C- level are 45%. An Electronic Performance 

Support System (EPSS) was designed and implemented to help the B-level and C-level people improve their 
performance. (See Leighton, 1996 for a description of the EPSS model.) The JIT Learning EPSS was designed and 
developed by Infomark Software Corporation with CGM Communications and assistance from several graduate 
students in the SFSU ITEC Department in the first half of 2001 . 

The JIT Learning EPSS was implemented with 15 learners in Fall 2001 . For this class the grade distribution was 
53% in the A-level, 20% in the B-level, and 27% in the C-level. When compared with prior classes, all of the B- 
level performers improved to A-level and half of the C-level improved to B-level. A formative evaluation of the 
system was conducted in the winter of 2002. The results suggest that the remaining C-level performers do not have a 
sufficient mental model of the design task. The next version of the software will focus on helping users develop and 
assess their mental models before they begin the design task. 

Performance Support Systems for Complex Problem-Solving 

Designing a performance improvement intervention is a moderately -structured complex problem-solving task 
(Foshay, 1997; Jonassen, 1997). Typically, a performance problem can be defined clearly, it is clear when the 
performance problem is solved, and there are multiple ways to design a correct solution. 

There are numerous examples of performance support systems used to teach complex problem-solving tasks. There 
is a subset of cases where the user performs a task primarily involving design, analysis, and/or planning and the 
output of the task is to produce a deliverable document. 

■ Broadcast News allows high school students to write news story on real current events (Schank & Kass, 1996). 
The system includes a tool for writing stories, background information on a story, and a library of information 
resources. Video advising is provided based on the concept of goal-based scenarios. 

■ Project-Based Learning Support System (PBLSS) allows high school students to plan and conduct a math or 
scientific inquiry project (Laffey, Musser, andTupper, 1998). The system includes guidelines for organizing a 
project, a tool for writing a project plan, and the ability to publish a project plan as an HTML document. PBLSS 
is designed to support project-based learning, where students work in groups to define and investigate their own 
project ideas. 

■ Air Campaign Planning Advisor (ACPA) allows U.S. Air Force officers to author air campaign plans to attack 
military targets (Johnson, Bimbaum, Bareiss, and Hinrichs, 2000). A seven-step model is used to structure the 
process. Information resources, including video-based advising, are organized around questions to address in 
producing an air campaign plan. 

■ Tactical Readiness Instruction, Authoring, and Delivery (TRIAD) allows U.S. Navy officers to create tactical 
documents that are used to share information on tactical problems and solutions (Hannafm, Hill, and McCarthy, 
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2001). TRIAD includes an authoring tool for producing multimedia documents, An interview process is used to 
lead the user through writing the various sections of a document. 

A common theme among these systems is support for two major activities. First, users must perform the complex 
problem-solving task that involves design, analysis, and/or planning. Second, users must perform the task of 
producing a deliverable document that contains the "output" of the problem-solving task. 

Underlying the design of these systems is the concept of situated cognition and the Cognitive Apprenticeship 
framework (Brown, Collins, and Duguid, 1989; Collins, Brown, and Holum, 1991). The EPSS contains a tool for 
producing the deliverable and uses a series of guidelines and questions to scaffold task performance. Information 
resources are organized around sub-tasks in order to model performance. Coaching is implemented in several ways: 
the process is structured through an explicit task model, the guidelines and questions provide implicit structuring as 
performance occurs, and video-based advising appears in the context of authentic task performance. 

The Just-In-Time Learning EPSS 

The Just-In-Time Learning EPSS uses a six-step process to design a performance improvement intervention. The 
first five steps lead the learner through describing the problem situation and explaining the design of the proposed 
intervention. The last step reviews the design strategy of the intervention. This last step is used to support 
articulation and reflection on the current design effort, and to explore new design ideas for future efforts. 

The format of a typical deliverable produced by the EPSS is like the format of this article. A paper is generated as a 
series of paragraphs with appropriate subheadings in HTML. The formatting of the document is simple, as the 
emphasis is on the content not on presentation. Of course, a user can import the HTML into a word processing 
program to improve the formatting. 



Structuring the Performance and Learning Process 

The opening web page for the EPSS shows the status of work on each step, or deliverable document. The web page 
for one of the deliverables is shown in Figure 1 . The header across the top of the page reiterates all the steps in the 
process. Access to performance and learning resources are organized around producing the deliverable. 
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Figure 1 - Guidelines and Resources for Producing a Deliverable Document 



Resources for a deliverable are organized into three categories, as noted below. The user clicks a S symbol to see 
the questions under a guideline or the documents for a type of resource. 

■ Guidelines and specific questions to answer in producing the deliverable document 
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■ Performance Resources- Advice on key design decisions, example documents, and information resources 
in the form of handout documents and articles to read 

■ What You Need to Know- These learning topics represent pre-requisite knowledge required in order to 
understand howto use the resources and produce the deliverable. 

The items under ”What You Need to Know" correspond to the learning objectives of the JIT Learning course. A 
learning topic has information resources like article readings, text discussions, and handout documents. It also 
includes exercises (self-graded) and assignments (graded by the instructor) for practice. These learning topics allow 
learners to review the information provided in class. 

Scaffolding Task Performance 

The combination of guidelines and questions structure the task of writing each deliverable document. By clicking on 
any question (or the question number following "Continue on Questions to Answer at:") the user can enter an 
answer for a question 



Figure 2 - Entering the Answer for a Question 



as shown in Figure 2. 
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The software simplifies the task of writing a document by prompting a user to answer a series of questions. When 
the user clicks "Save" an answer is saved in a relational database. Then the user is prompted to answer the next 
question. A progress meter showing how many questions have been answered for the deliverable appears on both 
the Deliverable and Question pages. 

At any time the user may click "View My Paper" to produce the output document. All of the individual answers are 
compiled into a series of paragraphs. Users can save documents in HTML format and import them into a word 
processing program for additional formatting or revision. 

Another scaffold in the system concerns the ability to prepare an outline before writing a paper (via the "Paper or 
Outline" button in Figure I). In outline mode, the user's answers are shown side-by-side in a two-column table with 
the questions. The learner can submit an outline version, get feedback from the instructor, and then revise the outline 
into a narrative paper. 

There are multiple levels to fading these scaffolds. The learner can skip producing an outline and move directly to 
writing a paper. The learner does not have to use the question and answer page as the writing tool. The Deliverables 
page could serve as an external support facility and the user could work entirely in a word processing program. By 
design, the fading of the scaffolds is controlled by the user. 
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Coaching and Advising 

Video-based advising takes the form of 45 to 90 second video clips with advice on the most frequently encountered 
design problems for a deliverable. This advising is in the form of a story about a design problem. Most of the videos 
feature learners who have completed the course telling their own painful stories. They describe a problem situation 
and then discuss their solution. There are also clips featuring the advice of the instructor. The instructor describes a 
symptom, the underlying problem, and advice on how to solve that problem. For example, one video clip explains 
the trap of starting to design the solution before the problem is fully defined. 

Formative Evaluation Results 



A formative evaluation of the EPSS was conducted in the winter of 2002. Data was collected from twelve 
participants with a 70-point questionnaire, from database log files with details on how the software was used, and 
with follow-up interviews. 

The average amount of time spent writing papers with the system was 927 minutes, or about 15>2 hours. When the 
participants are ranked by final grade, the top third spent an average of 975 minutes, the middle third spent 494 
minutes, and the bottom third spent 1,313 minutes. The following chart shows the average amount of time spent on 
each of the four major papers (Deliverables) by group. The differences are quite obvious between the three groups. 
The top group put most of their effort into the problem description, while the bottom group spent most of their time 
on describing the solution (the Components and Roles and Case Scenario papers). The middle group appears to be 
quite efficient in using their time. In contrast, given the low quality of their design efforts, the bottom group spent a 
lot of unproductive time in devising a solution. 



On the questionnaires, the bottom group scored themselves higher than the top group in using the feedback from the 
instructor to improve their project designs. However, their final grades were lower primarily because they did not 
correct the design flaws identified in feedback on their papers. The Case Scenario papers by the bottom group 
revealed a weak understanding of the task being performed by their target users. Surprisingly, the bottom group 
claimed to have more experience doing task analysis than the top group. Even though this topic is covered in a 
prerequisite course on the ISD model, the bottom group appears to have a weak understanding of task analysis. This 
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limits their ability to effectively use the instructor's feedback in defining problems and designing solutions. 

Implications and Future Work 



The approach of using guidelines and questions to scaffold the task of writing complex papers was successful in 
helping improve the overall performance of a class. However, there is still a group at the bottom of the class that 
needs more help than the EPSS provided. They spend too little time in defining the problem situation and too much 
time in creating a solution. At the same time, they do not make effective use of the instructor's feedback in 
correcting design flaws. 

This suggests that users who perform poorly have weak mental models of the design task. In the next version of the 
JIT Learning EPSS we will add functions to help users develop and assess their own mental models before they 
begin producing a deliverable paper. This will include more emphasis on using the analysis of a target user's task to 



5 



better understand the problem domain. We will also add a feedback-and-response function so that users explicitly 
acknowledge the instructor’s feedback and articulate how they will correct design flaws. 
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